Papers with probe complexity
Low-Complexity Probing via Finding Subnetworks (2021.naacl-main)
Copied to clipboard
| Challenge: | Existing approaches to probing neural networks for linguistic properties are to train a shallow multi-layer perceptron (MLP) on top of the model's internal representations. |
| Approach: | They propose a subtractive pruning-based probe where they find an existing subnetwork that performs the linguistic task of interest. |
| Outcome: | The proposed probe achieves higher accuracy on pre-trained models and lower accuracy on random models, and better learning on its own. |
Pareto Probing: Trading Off Accuracy for Complexity (2020.emnlp-main)
Copied to clipboard
| Challenge: | Neural networks are a pillar of modern NLP systems, but their inner workings are poorly understood. |
| Approach: | They propose a probe metric that reflects the trade-off between probe complexity and performance: the Pareto hypervolume. |
| Outcome: | The proposed probe metric conforms to accepted rankings among contextual representations, and is more complex than other probe tasks. |